Add read_openephys_binary and catalogue-based probe construction for OpenEphys#406
Add read_openephys_binary and catalogue-based probe construction for OpenEphys#406h-mayorquin wants to merge 5 commits intoSpikeInterface:mainfrom
read_openephys_binary and catalogue-based probe construction for OpenEphys#406Conversation
| np_probe_info = np_probes_info[probe_idx] | ||
| np_probe = np_probes[probe_idx] | ||
| probe = np_probe_info.get("probe") | ||
| def read_openephys_binary( |
There was a problem hiding this comment.
We can have a single read_openephys function, with an optional oebin_file argument and two private functions for the actual reading
| electrode_index_to_column = {ei: col for col, ei in enumerate(oebin_electrode_indices)} | ||
| device_channel_indices = np.zeros(num_contacts, dtype=int) | ||
| for i, contact_id in enumerate(probe.contact_ids): | ||
| electrode_index = int(contact_id.split("e")[-1]) |
There was a problem hiding this comment.
I think there's a mismatch between our multi-shank probe electrode ids and the electrode ids in the oebin file which causes problems here.
If we have multiple shanks, our electrode ids are e.g.
s0e0, s0e1, s0e2...
s1e0, s1e1, s1e2...
s2e0, s2e1, s2e2...
s3e0, s3e1, s3e2...
But the oebin file has a unique electrode id for each electrode: i.e. for my NP2 4-shank probes, there are 5116 channels and the electrode ids in the oebin file go from 0 to 5116.
I think it's ok, because the fallback does the right thing. But it makes me think we're missing something.
Here's the settings and structure files for out probes (need to json the structure file for github to accept it):
3 participants
This adds
read_openephys_binary(settings_file, oebin_file, stream_name), a new reader for Open Ephys binary format recordings that uses bothsettings.xml(probe geometry) andstructure.oebin(binary data layout) to build probes. It also refactorsread_openephysto use catalogue-based probe construction, fixing contact_id consistency with SpikeGLX. See #405 for the broader discussion on separating probe construction from wiring.Both readers now follow the same pattern as
read_spikeglx(introduced in PR #383): build the full probe from the catalogue viabuild_neuropixels_probe, then combine it with data from the source files to determine which electrodes are active and how they map to the data. For SpikeGLX, the source file is the.metafile (IMRO table). For Open Ephys, thesettings.xmlprovides electrode positions (viaELECTRODE_XPOS/ELECTRODE_YPOS) or electrode indices (viaSELECTED_ELECTRODES), which are matched against the catalogue probe geometry to select the active contacts.read_openephys_binaryadditionally reads thestructure.oebinto setdevice_channel_indices. As proposed in SpikeInterface/spikeinterface#4435, the oebin contains per-channelelectrode_indexmetadata (added in neuropixels-pxi plugin v0.5.0, January 2023) that maps each binary file column to its electrode index. When available and non-zero,read_openephys_binaryuses this for wiring. For older recordings whereelectrode_indexis missing or all zeros, it falls back to identity wiring, which is correct because the binary.datfile is always written in channel-number order (as confirmed by Josh Siegle). The existingread_openephysis kept for backward compatibility and always sets identity wiring since it has no oebin.Because both readers now derive
contact_idsfrom the catalogue probe, this also fixes #394 (OpenEphyscontact_idsinconsistent with SpikeGLX). The previous code reverse-engineered electrode IDs from positions, which produced different IDs than SpikeGLX for probes where OpenEphys positions don't exactly match the canonical geometry. @chrishalcrow identified the affected datasets in the existing test data (OE_Neuropix-PXI-NP-Ultra,OE_6.7_enabled_disabled_Neuropix-PXI,OE_Neuropix-PXI-QuadBase). A new consistency test verifies that for these and other datasets, thecontact_idsfromread_openephysare a subset of the catalogue probe'scontact_ids.Other changes:
read_openephys_binaryis annotated withplugin_channel_key(the XML CHANNELS key, e.g. "CH0", "CH29"), distinct from neo'schannel_namewhich comes from the oebin and uses different naming conventions.settings.xml+structure.oebinfiles.